(torchx/local_scheduler) Use os.kill instead of os.killpg when sending SIGTERM to the replica pid. Add runner.wait() for torchx.runner.test.api_test#test_empty_session_id to gracefully wait for the replicas to finish running #1063
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…g SIGTERM to the replica pid. Add runner.wait() for torchx.runner.test.api_test#test_empty_session_id to gracefully wait for the replicas to finish running Summary: Fixes macos unittest failures: https://github.com/pytorch/torchx/actions/runs/14844253481/job/41674243877 When looking into the test failure I noticed two things: 1. `local_scheduler` was trying to SIGTERM the process group by passing the replica's pid: `os.killpg(replica.pid, signal.SIGTERM)` . Changed to call `os.kill`. (note that `os.killpg` is not available on iOS which is why the test was failing). 2. The `torchx.runner.test.api_test.test_empty_session_id()` test case doesn't wait for the `echo` test command to finish hence there was a race condition where in certain cases the runner's `__exit__()` SIGTERMs the replica pids but since the `local_scheduler` was (wronfully) using `os.killpg` not `os.kill` it threw an uncaught error in iOS. Differential Revision: D74197282
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D74197282 |
|
duplicate of #1062. Closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Fixes macos unittest failures: https://github.com/pytorch/torchx/actions/runs/14844253481/job/41674243877
When looking into the test failure I noticed two things:
local_schedulerwas trying to SIGTERM the process group by passing the replica's pid:os.killpg(replica.pid, signal.SIGTERM). Changed to callos.kill. (note thatos.killpgis not available on iOS which is why the test was failing).The
torchx.runner.test.api_test.test_empty_session_id()test case doesn't wait for theechotest command to finish hence there was a race condition where in certain cases the runner's__exit__()SIGTERMs the replica pids but since thelocal_schedulerwas (wronfully) usingos.killpgnotos.killit threw an uncaught error in iOS.Differential Revision: D74197282